Reputation in the Academic World

This paper proposes a computational model based on peer reviews for assessing the reputation of researchers and research work. We argue that by relying on peer opinions, we address some of the pitfalls of current approaches for calculating the reputation of authors and papers. We also introduce a much needed feature for review management: calculating the reputation of reviews and reviewers. 1 MOTIVATION With open access gaining momentum, open reviews becomes a more persistent issue. Current institutional and multidisciplinary open access repositories lack the quantitative assessment of the hosted research items that will facilitate the process of selecting the most relevant and distinguished content. Common currently available metrics, such as number of visits and downloads, do not reflect the quality of a research product, which can only be assessed directly by peers offering their expert opinion together with quantitative ratings based on specific criteria. The articles published in the Frontiers book [1] highlight the need for open reviews. To address this issue we develop an open peer review module, the Academic Reputation Model (ARM), as an overlay service to existing institutional or other repositories. The model calculates the reputation of authors, reviewers, papers, and reviews, by relying on peer opinions. We argue that this addresses some of the pitfalls of current approaches for calculating the reputation of authors and papers. It also introduces a much needed feature for review management, and that is calculating the reputation of reviews and reviewers. 2 ARM: ACADEMIC REPUTATION MODEL 2.1 Data and Notation To compute its reputation measures, ARM requires a reputation data set (which should be extracted from existing repositories) specified as the tuple 〈P,R,E,D, a, o, v〉, where • P = {pi}i∈P is a set of papers (e.g. DOIs). • R = {rj}j∈R is a set of researcher names or identifiers (e.g. the ORCHID identifier). • E = {ei}i∈E ∪ {⊥} is a totally ordered evaluation space, where ei ∈ N \ {0} and ei < ej iff i < j and ⊥ stands for the absence of evaluation. We suggest the range [0,100], although any other range may be used, and the choice of range will not affect the performance. • D = {dk}k∈K is a set of evaluation dimensions, such as originality, technical soundness, etc. • a : P → 2 is a function that gives the authors of a paper. 1 Artificial Intelligence Research Institute (IIIA-CSIC), Barcelona, Spain, email: {nardine, sierra}@iiia.csic.es • o : R × P × D × T ime → E, where o(r, p, d, t) ∈ E is a function that gives the opinion of a reviewer, as a value in E, on a dimension d of a paper p at a given instant of time t. • v : R×R× P × T ime → E, where v(r, r′, p, t) = e is a function that gives the judgement of researcher r over the opinion of researcher r′, on paper p as a value e ∈ E. Therefore, a judgement is a reviewer’s opinion about another reviewer’s opinion. 2.2 Reputation of a Paper The reputation of a paper is a weighted aggregation of its reviews, where the weight is the reputation of the reviewer (Section 2.4).